Ok it is a first tentative. I could collect some feedback

@shelhamer Is this make sense?
@sguada It this semantically conflict with your Multi label accuracy use of label blob channels?

I still don't see the logic behind this. Could you pose an example ?

On Saturday, July 26, 2014, bhack notifications@github.com wrote:

@shelhamer https://github.com/shelhamer Is this make sense?
@sguada https://github.com/sguada It this semantically conflict with
your Multi label accuracy use of label blob channels?

—
Reply to this email directly or view it on GitHub
#759 (comment).

Sergio

There are some cases where the source has N multi class label (on a single image). I can split the convolutional layer for feed N inner product layers (one per single label) and then I can use a N softmaxloss for every inner product. But I need to use the new slice layer on label blob to create N different top blob label. On test I want to calc global accuracy so I need to increment accuracy when N softmax layer predict exactly N label. So I need with a concat layer to aggregate back sliced labels and extend accuracy layer to support multiple labels aggregated in channels.

@sguada @shelhamer Do you need some other info?

@jeffdonahue if you are still active today can you take a look to this? I've some free time tomorrow and I want to fix it if it has any useful goal.

Sorry, I'm not sure I understand how this is different from MultiLabelAccuracy by @sguada. Is this a generalization in that you're doing N 1-of-M predictions per datum, whereas MultiLabelAccuracy is for N binary predictions (1-of-2), and the current AccuracyLayer is for a single 1-of-M prediction?

Yes exactly. I think this could be still "emulated" with @sguada Multi label "one hot" encoding using -1 0 1 to emulate every N softmax/softmax loss and then concat to feed Multilab accurancy. But i.e. we cannot use standard softmax/softmax loss to every single N and use standard accuracy. This try to enable this last behaviour.

I guess there could be a case of use, but I think it is not very common. So
not sure worth the effort, but if you want to do it and add real example
where the usage is clearly illustrated, go ahead. I'm a bit concern that
different set of labels would have different number of labels, what would
complicate the definition of the layer. If I were to implemented it I would
just group a set of softmax and accuracy layers.

As you noted one could either use several Softmax and Accuracy Layers, or
Multi-label Layer could be used.

Sergio

2014-07-30 1:02 GMT-07:00 bhack notifications@github.com:

Yes exactly. I think this could be still "emulated" with @sguada
https://github.com/sguada Multi label "one hot" encoding using -1 0 1
to emulate every N softmax/softmax loss and then concat to feed Multilab
accurancy. But i.e. we cannot use standard softmax/softmax loss to every
single N and use standard accuracy. This try to enable this last behaviour.

—
Reply to this email directly or view it on GitHub
#759 (comment).

@sguada An example that we already discussed in another thicket is partially in the schema at page 13 of this paper at the state of the art on SVHN dataset.
I cannot directly use several Softmax and a standard Accuracy Layers for achieve a global accuracy because i have i.e. a single accuracy for a single digit. This is a multi-target multi-class problem and i need to increment the global accuracy only when every Softmax predict the right digits. Can this be achieved in other way without using "one hot" emulation of your PR?

I guess you can go ahead and try to implement such layer, but it seems a
bit overkill create one layer, just for one problem. Although there could
be other uses. Also having different lengths would complicate using several
softmax layers.

I don't see any problem using "one hot" emulation for that either.

Sergio

2014-07-30 9:16 GMT-07:00 bhack notifications@github.com:

@sguada https://github.com/sguada An example that we already discussed
in another thicket is partially in the schema at page 13 of this paper
http://arxiv.org/pdf/1312.6082v4.pdf at the state of the art on SVHN
dataset.
I cannot directly use several Softmax and a standard Accuracy Layers for
achieve a global accuracy because i have i.e. a single accuracy for a
single digit. This is a multi-target multi-class problem and i need to
increment the global accuracy only when every Softmax predict the right
digits. Can this be achieved in other way without using "one hot" emulation
of your PR?

—
Reply to this email directly or view it on GitHub
#759 (comment).

@sguada This PR tries to modify the actual accuracy layer it doesn't create a new layer. I'm waiting also for @shelhamer and @jeffdonahue feedbacks. If generally in this cases we want to handle this network design with "one hot" softmax emulation + label split + "innerproduct and label concat" to feed multilabel accurancy in #523 we can close this PR.

I looked a little bit more in Multilabel accuracy in #523 and it doesn't work as a global accuracy on multitarget when we need to increment accuracy only when all target are predicted correctly and for one hot encoding use has still the problem of argmaxing probabilities. If we want to support a global accuracy from multiple loss we need to use something like this PR or extend the accuracy layer to support a vector of bottom blob of argmax=labels.

I think having a global loss from multiple losses could be useful. So you
can go ahead and implement an example with the new layer, and add some docs
on how to use it.

Sergio

2014-08-05 14:04 GMT-07:00 bhack notifications@github.com:

I looked a little bit more in Multilabel accuracy in #523
#523 and it doesn't work as a global
accuracy on multitarget when we need to increment accuracy only when all
target are predicted correctly and for one hot encoding use has still the
problem of argmaxing probabilities. If we want to support a global accuracy
from multiple loss we need to use something like this PR or extend the
accuracy layer to support a vector of bottom blob of argmax=labels.

—
Reply to this email directly or view it on GitHub
#759 (comment).

Ok Sergio but I'm not talking about global loss from multiple loss. I'm talking of a global accuracy from multiple independent loss and labels. This PR extend Caffe accuracy layer try to cover this case. I'm using channels to concat labels. I could simplify this PR so that we could use in a standard way where argmax concat on channels == labels concat on channels. If we want to cover global accuracy with this configuration.

I think the code you have right now, it is too specific. It assumes that all the set of labels have the same number labels. Therefore I don't see applicability to any other problem.
Also I think it is more relevant to modify the loss rather that the accuracy, since the accuracy it is only informative but not useful for training.

I was not directly interested in modify the loss but i could try to work on this. What is your idea? If we have N single loss that try to optimize a branch of the DAG and contribute backward to a common parent (and this is already supported in Caffe) what kind o recombination of the single branches for a global loss are you thinking about? At informative level we need also a global accuracy so how we solve that? If as you tell is not generic to have argmax = labels how do you will handle the informative level?

I see some general material here

@shelhamer You have close the related ticket but what we want to do with this PR?

Closing since the dev branch is deprecated. Please send PRs to master.